This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLf ttlAOlPS 

Defective images within this document are ace«rat@f@|»eS@^tois efttoaignd 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BIACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES- 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXfflBFT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: „_____________^^ 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, pSense do. not report these proiiems to 
the IFW Image Problem Mailbox. 



United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



26694 7590 10/21/2004 

VENABLE, BAETJER, HOWARD AND CIVILETTI, LLP 
P.O. BOX 34385 

WASHINGTON, DC 20043-9998 



APPLICATION NO. 


FILING DATE\^ 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/694,712 


1 10/24/2000 


Alan J. Lipton 


37112-164994 


3252 



EXAMINER 



LE, VU 



ART UNIT 



PAPER NUMBER 



2613 

DATE MAILED: 10/21/2004 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO 90C (Rev 10/03) 



Office Action Summary 


Application No. 

09/694,712 


Applicant(s) 

LIPTON, ALAN J. 


Examiner 

Vu Le 


Art Unit 

2613 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 



Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )Kl Responsive to communication(s) filed on 23 February 2001 . 
2a)\Z\ This action is FINAL. 2b)KI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) [x] Claim(s) 1-58 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) [X] Claim(s) 1-58 is/are rejected. 

7) Q Claim(s) is/are objected to. 

80 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9)Q The specification is objected to by the Examiner. 

10O The drawing(s) filed on is/are: a)\Z\ accepted or b)d objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 1 19 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 0 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach m en t(s) 

1 ) [3 Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Pa P er No(s)/Mail Date. . 

3) H Information Disclosure Statement(s) {PTO-1449 or PTO/SB/08) 5 ) D Notice of Informal Patent Application (PTO-152) 

Paper No(s)/Mail Date . 6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 101304 



Application/Control Number: 09/694,712 
Art Unit: 2613 



Page 2 



DETAILED ACTION 
Claim Rejections - 35 USC § 102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent. 

(e) the invention was described in- 

(1) an application for patent, published under section 122(b), by another filed in the United States before the 
invention by the applicant for patent; or 

(2) a patent granted on an application for patent by another filed in the United States before the invention by 
the applicant for patent, except that an international application filed under the treaty defined in section 351(a) 
shall have the effects for the purposes of this subsection of an application filed in the United States only if the 
international application designated the United States and was published under Article 21(2) of such treaty in 
the English. 

2. Claims 1-5, 16-20, 29-42, 51-58 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Bulman et al, US Publication 2003/0051255. 

Re claim 1, Bulman et al discloses a method (figs. 3, 19) comprising the steps of: 
extracting an object of interest from a video stream (21 1-212, para. 0265-0266); 

analyzing said object from said video stream to obtain an analyzed object (213, para. 

0267); manipulating said analyzed object to obtain a synthetic character (215-216, para. 

0569-0270); and assembling a virtual video using said synthetic character (217, para. 

0271, also para. 0167-0168). 

Re claim 2, the method of claim 1 , wherein said step of extracting comprises the 

step of employing a stochastic background modeling technique to detect said object in 

said video stream. (See para. 0266). 
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Re claim 3, the method of claim 1 , wherein said step of extracting comprises the 
step of employing a motion detection technique to detect said object in said video 
stream. (See para. 0266). 

Re claim 4, the method of claim 1 , wherein said step of extracting comprises the 
step of employing a segmentation technique to detect said object in said video stream. 
(See para. 0266, also para. 0036). 

Re claim 5, the method of claim 1 , wherein said step of extracting comprises the 
step of extracting background components from said video stream. (See para. 0266). 

Re claim 16, the method of claim 1, wherein said step of manipulating is directed 
by a user. (See para. 0165, 0167). 

Re claim 17, the method of claim 1, wherein said step of manipulating is directed 
by a computation engine. (See para. 0206-0208). 

Re claim 18, the method of claim 1, wherein said synthetic character is a hybrid 
based on said object of interest and computer-generated imagery. (See figs. 12A-12E, 
para. 0233). 

Re claim 19, the method of claim 1 , further comprising the step of providing a 
second synthetic character generated by a computer graphics engine, and wherein said 
step of assembling comprises the step of assembling said virtual video using said 
synthetic character and said second synthetic character. (See para. 0245-0247, in 
these segments, a library of synthetic images is provided to produce a virtual video). 
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Re claim 20, the method of claim 1 , wherein said step of assembling comprises 
the step of inserting said synthetic character into said video stream. (See para. 0244- 
0247). 

Re claim 29, the method of claim 1 , wherein said step of assembling comprises 
the step of removing said synthetic character from said video stream. (See para. 0082, 
this segment discusses the ability to customize synthesized content so that arbitrary 
substitutions, additions and alterations may applied). 

Re claim 30, the method of claim 29, wherein said step of removing comprises 
the step of filling in pixels in frames of said video stream with pixels from a background 
model for said synthetic character removed from said video stream. (See para. 0207- 
0209). 

Re claim 31 , the method of claim 29, wherein said step of removing comprises 
the step of repairing at least one of an uncovered background, a foreground object, and 
another synthetic character. (See para. 0082, this segment discusses the ability to 
customize synthesized content so that arbitrary substitutions, additions and alterations 
may applied. In this context, the capability of repairing at least one of an uncovered 
background, a foreground object, and another synthetic character would have been 
inherent through the process of customization). 

Re claim 32, the method of claim 1 , further comprising the step of determining 
functional areas within said video stream. (See fig. 19: 212, para. 0265-0269, in this 
segment, the step of extracting object(s) from a video stream would have inherently 
include the step of determining functional areas within said video stream). 



Application/Control Number: 09/694,712 Page 5 

Art Unit: 2613 

Re claim 33, the method of claim 1, further comprising performing the method of 
claim 1-for a plurality of objects of interest in said video stream. (The limitations have 
been analyzed and rejected w/r to claim 32. In this segment, multiple objects are 
considered). 

Re claim 34, the method of claim 1, wherein said steps of extracting, analyzing, 
manipulating, and assembling are performed in real time. (See para. 0270, see also 
para. 0049). 

Re claim 35, the method of claim 1 , wherein at least one of said steps of 
extracting, analyzing, manipulating, and assembling is performed in non-real time. (The 
limitations have been analyzed and rejected w/r to claim 1, also see para. 0030, 0049, 
0100, in this segment, Bulman et al discloses the intent of real-time or near real-time 
processing, however, the process has the inherent capability to perform in non real-time 
as claimed). 

Re claim 36, the method of claim 1 , wherein said video stream comprises a 
background for a game. (See para. 0054, 0080). 

Re claim 37, the method of claim 1 , wherein said video stream comprises a 
simulation. (See para. 0082). 

Re claim 38, the method of claim 1, wherein said video stream comprises a 
teleconference. (See para. 0078). 

Re claim 39, the method of claim 1, wherein said video stream comprises a 
distance education presentation. (See para. 0078, this segment discusses on-line 
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multimedia presentation, which inherently is capable of distance education 
presentation). 

Re claim 40, a computer system to perform the method of claim 1 . (See fig. 3). 

Re claim 41 , a system comprising means for processing to perform the method 
of claim 1. (See fig. 3). 

Re claim 42, a computer-readable medium comprising software to perform the 
method of claim 1. (See fig. 3, also para. 0161-0163). 

Claim 51 recites "[A]" method comprising the steps of: extracting in real time a 
background model from a video stream; generating in real time a synthetic character; 
and assembling in real time a virtual video based on said background model and said 
synthetic character[.]" (The limitations have been analyzed and rejected w/rto claims 1- 
2). 

Re claim 52, a method as in claim 51, wherein said step of generating comprises 
generating said synthetic character using a computer graphics engine. (The limitations 
have been analyzed and rejected w/r to claims 1-2, 18-19). 

Re claim 53, a method as in claim 51 , further comprising the step of extracting in 
real time an object of interest from said video stream, and wherein said step of 
generating comprises generating said synthetic character using said object. (The 
limitations have been analyzed and rejected w/rto claims 1-2, 34). 

Re claim 54, a method as in claim 51 , further comprising the step of extracting in 
real time an object of interest from said video stream, and wherein said step of 
generating comprises generating said synthetic character using said object and a 
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computer graphics engine. (The limitations have been analyzed and rejected w/r to 
claims 1-2, 18-19, 34). 

Re claim 55, a method as in claim 51 , further comprising the step of identifying a 
functional area in said video stream, and wherein said step of assembling comprises 
assembling said virtual video based on said background model, said synthetic 
character, and said functional area. (The limitations have been analyzed and rejected 
w/r to claims 1-2, 32). 

Re claim 56, a computer system to perform the method of claim 51 . (The 
limitations have been analyzed and rejected w/r to claims 1-2, 40). 

Re claim 57, a system comprising means for processing to perform the method 
of claim 51 . (The limitations have been analyzed and rejected w/r to claims 1-2, 41 ). 

Re claim 58, a computer-readable medium comprising software to perform the 
method of claim 51 . (The limitations have been analyzed and rejected w/r to claims 1-2, 
42). 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 6-8 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Bulman et al as applied to claim 1 above and further in view of Lipton et al, 
"Moving Target Classification and Tracking from Real-time Video". 
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Re claim 6, Bulman et al discloses the method of claim 1 , wherein said step of 
extracting comprises the steps of: determining if a pixel in a frame of said video stream 
represents an object that is moving based on a stochastic model of a background scene 
in said frame (see analysis/rejection w/r to claim 2), but fails to disclose "clustering 
pixels in said frame that represent said object that is moving" as claimed. 

Lipton et al makes it well known and obvious extracting an object comprising the 
steps of: determining if a pixel in a frame of said video stream represents an object that 
is moving based on a stochastic model of a background scene in said frame; and 
clustering pixels in said frame that represent said object that is moving (para. 3: "Target 
Classification", in this segment, clustering is used to identify and classify moving 
object(s) from background clutter for target tracking application). 

Therefore, taking the combined teaching of Bulman et al and Lipton et al as a 
whole, it would have been obvious to augment the object extraction algorithm in Bulman 
et al to include clustering to identify and classify moving object(s) from background 
clutter as suggested in Lipton et al for the benefit of robust identification of objects of 
interest, rejection of background clutter, and continuous tracking of objects over a 
period of time despite occlusions, appearance changes and cessation of object motion 
(Abstract). 

Re claim 7, Bulman et al discloses object extraction (see analysis/rejection w/r to 
claim 1), but fails to disclose comprising the step of tracking said object. Lipton et al 
makes it well known and obvious extracting an object comprising the step of tracking 
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said object (para. 3: "Target Classification", in this segment, clustering is used to identify 
and classify moving object(s) from background clutter for target tracking application). 

Therefore, taking the combined teaching of Bulman et al and Lipton et al as a 
whole, it would have been obvious to modify Bulman et al to include object tracking as 
suggested by Lipton et al for the benefit of surveillance application when such 
application is desirable. 

Re claim 8, the method of claim 7, wherein said step of tracking comprises the 
steps of: identifying a position of said object in a frame of said video stream; identifying 
candidate objects in a next frame of said video stream; and comparing said object in 
said frame with candidate objects in said next frame to determine a next position of said 
object in said next frame. (See Lipton et al, fig. 7, para. 4: "Tracking", Motivation to 
combine has been established in claim 7). 

5. Claim 9-15 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Bulman et al as applied to claim 1 above and further in view of Lipton, "Virtual 
Postman - Real-Time, Interactive Virtual Video". 

Re claim 9, Bulman et al discloses analyzing said object after extraction (see 
analysis/rejection w/r to claim 1 ), but fails to disclose the step of analyzing comprises 
the step of determining a rigidity of said object. Lipton makes it well known and obvious 
the step of analyzing an extracted object to determine its rigidity (para. 5: "Character 
Rigidity", also fig. 4). 
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Therefore, taking the combined teaching of Bulman et al and Lipton as a whole, it 
would have been obvious to modify Bulman et al in which object analysis includes 
rigidity determination as taught by Lipton for the benefit of synthesizing characters to 
appear realistic so as to replace live characters in a real-time video stream. Also, this 
capability is beneficial for applications such as computer vision, image analysis, virtual 
video, and augmented reality (Lipton: Abstract). 

Re claim 10, the method of claim 9, wherein said step of determining said rigidity 
comprises the step of determining a residual flow for said object. (See Lipton, para. 5: 
"Character Rigidity"). 

Re claim 1 1 , Bulman et al discloses analyzing said object after extraction (see 
analysis/rejection w/r to claim 1 ), but fails to disclose the step of analyzing comprises 
the step of determining a periodic sequence corresponding to said object. Lipton makes 
it well known and obvious the step of analyzing comprises the step of determining a 
periodic sequence corresponding to said object (para. 6: "Determining a Periodic 
Sequence", also fig. 5). 

Therefore, taking the combined teaching of Bulman et al and Lipton as a whole, it 
would have been obvious to modify Bulman et al in which object analysis includes 
determining a periodic sequence corresponding to said object as taught by Lipton for 
the benefit of generating synthetic characters based on video imagery. In other words, 
to make the synthetic characters appear realistic (Lipton: para. 6: "Determining a 
Periodic Sequence", also fig. 5). 
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Re claim 12, the method of claim 11, wherein said periodic sequence represents 
one cycle of motion of said object over a series of frames, and wherein said periodic 
sequence comprises, for each frame of said set of frames, a visual appearance of said 
object and a frame-to-frame displacement of said object. (Lipton: para. 6: "Determining 
a Periodic Sequence", also fig. 5). 

Re claim 13, the method of claim 11, wherein said step of determining said 
periodic sequence comprises the steps of: collecting a set of visual templates of said 
object from a series of games of said video stream, said set of visual templates 
comprising at least one complete period of motion of said object; and matching a 
present visual template of said object with each visual template of said set of visual 
templates to determine a starting point of said period of motion of said object. (Lipton: 
para. 6: "Determining a Periodic Sequence", also fig. 5). 

Re claim 14, the method of claim 13, wherein said step of matching comprises 
the steps of: determining a convolution of said present visual template with each visual 
template of said set of visual templates; and selecting said starting point of said period 
of motion based on a minimum of said convolution. (Lipton: para. 6: "Determining a 
Periodic Sequence", also fig. 5). 

Re claim 1 5, the method of claim 1 , wherein said step of analyzing comprises the 
step of determining a periodic sequence corresponding to said object and at least one 
parameter describing an appearance of said object, and the method further comprising 
the step of transmitting or storing said periodic sequence and said at least one 
parameter. (Lipton: para. 7, 7.1 and 8, in these segments, discussion of using periodic 
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sequences of templates and velocities i.e. parameter to be captured and then replayed 
synthetically in arbitrary places and at arbitrary times from virtual video stream is 
indicative of the inherent need to store and transmit during real-time application). 

6. Claims 21-28 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Bulman et al as applied to claims 1, 20 above and further in view of Lipton, 
"Virtual Postman - Real-Time, Interactive Virtual Video". 

Re claim 21 , Bulman et al discloses inserting synthetic character into the video 
data stream (see analysis/rejection w/r to claim 20), but fails to disclose that the 
synthetic character is inserted based on rigidity and periodicity of said synthetic 
character. Lipton makes it well known and obvious synthetic character is inserted 
based on rigidity and periodicity of said synthetic character (para. 2: "A Virtual Video 
Architecture", this segment discusses several situations when it is necessary to insert 
synthetic characters into the virtual video stream in which rigidity and periodicity of a 
character are taken into account). 

Therefore, taking the combined teaching of Bulman et al and Lipton as a whole, it 
would have been obvious to modify Bulman et al so that the step of inserting synthetic 
character into a video stream is based on rigidity and periodicity of said synthetic 
character. The advantage of this modification would be the ability to identify and 
categorize different object motion, for example, human or animal motion as opposed to 
motion of vehicle in a video scene. 
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Re claim 22, the method of claim 20, wherein said step of inserting comprises the 
steps of selecting a starting frame in said video stream and a position within said 
starting frame for inserting said synthetic character; and inserting a periodic sequence 
corresponding to said synthetic character beginning in said starting frame and at said 
position. (Lipton: para. 6: "Determining a Periodic Sequence", also fig. 5). 

Re claim 23, the method of claim 22, wherein said periodic sequence represents 
one cycle of motion of said synthetic character, and wherein said periodic sequence 
comprises, for each frame of said set of frames, a visual template of said synthetic 
character and a frame-to-frame displacement of said synthetic character. (Lipton: para. 
6: "Determining a Periodic Sequence", also fig. 5). 

Re claim 24, the method of claim 23, wherein said step of inserting further 
comprises the step of multiplying each visual template by a scale factor to adjust a size 
of said synthetic character. (Lipton: para. 6: "Determining a Periodic Sequence", also 
fig. 5, also para 7: "Synthesizing Characters"). 

Re claim 25, the method of claim 23, wherein said step of inserting further 
comprises the step of multiplying each frame-to-frame displacement by a time factor to 
adjust a speed of motion of said synthetic character. (Lipton: para. 6: "Determining a 
Periodic Sequence", also fig. 5, also para 7: "Synthesizing Characters"). 

Re claim 26, the method of claim 23, wherein said step of inserting further 
comprises the step of applying a flip operator to each visual template to reverse a 
direction of motion of said synthetic character relative to a direction of motion of said 
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synthetic character in said periodic sequence. (Lipton: para. 6: "Determining a Periodic 
Sequence", also fig. 5, also para 7: "Synthesizing Characters"). 

Re claim 27, the method of claim 20, wherein said synthetic character is 
occluded in said video stream. (Lipton: para. 7.1: "Synthesising Occluded Characters"). 

Re claim 28, the method of claim 20, wherein said step of inserting comprises the 
step of modifying at least one of appearance, scale, position, speed, direction of motion, 
and timing, prior to insertion of said synthetic character into said video stream. (Lipton: 
para. 6: "Determining a Periodic Sequence", also fig. 5, also para 7: "Synthesizing 
Characters"). 

7. Claim 43, 45-50 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Bulman et al in view of Lipton et al, "Moving Target Classification and 
Tracking from Real-time Video". 

Claim 43 recites "[A] method comprising the steps of: obtaining a video stream as 
a setting for one of a video game, a simulation, a teleconference, and a distance 
education presentation; tracking a moving object in said video stream; analyzing said 
moving object to obtain an analyzed moving object; generating a synthetic character 
based on said analyzed moving object; and assembling a visual video based on said 
synthetic character and said video stream[.]". (The limitations have been analyzed and 
rejected w/r to claims 1 , 7, 37-39). 

Re claim 45, the method of claim 43, wherein said step of generating is in 
response to a user of one of said video game, said simulation, said teleconference, and 
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said distance education presentation. (The limitations have been analyzed and rejected 
w/r to claims 1,7,16, 37-39). 

Re claim 46, the method of claim 43, wherein said step of assembling comprises 
the step of inserting said moving object into said virtual video based on said synthetic 
character. (The limitations have been analyzed and rejected w/r to claims 1, 7, 20, 37- 
39). 

Re claim 47, the method of claim 43, wherein said step of assembling comprises 
the step of removing said moving object from said virtual video based on said synthetic 
character. (The limitations have been analyzed and rejected w/r to claims 1, 7, 29, 37- 
39). 

Re claim 48, a computer system to perform the method of claim 43. (The 
limitations have been analyzed and rejected w/r to claims 1, 7, 37-39, 40). 

Re claim 49, a system comprising means for processing to perform the method 
of claim 43. (The limitations have been analyzed and rejected w/r to claims 1 , 7, 37-39, 
41). 

Re claim 50, a computer-readable medium comprising software to perform the 
method of claim 43. (The limitations have been analyzed and rejected w/r to claims 1, 

7, 37-39, 42). 

8. Claim 44 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Bulman et al in view of Lipton et al, "Moving Target Classification and Tracking 
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from Real-time Video" as applied to claim 43 above and further in view of Lipton, 
"Virtual Postman - Real-Time, Interactive Virtual Video". 

Re claim 44, the method of claim 43, wherein said step of analyzing comprises 
the step of representing said moving object by a periodic sequence. (The limitations 
have been analyzed and rejected w/r to claims 1 , 7 and 11). 
9. Claims 1-58 are rejected under 35 U.S.C. 102(a) as being anticipated by 
Lipton, "Virtual Postman - Real-Time, Interactive Virtual Video". 

Re claim 1, Lipton discloses a method (fig. 1) comprising the steps of: extracting 
an object of interest from a video stream (fig. 1 , para. 2: A Virtual Video Architecture); 
analyzing said object from said video stream to obtain an analyzed object (fig. 1 f para. 2: 
A Virtual Video Architecture); manipulating said analyzed object to obtain a synthetic 
character (fig. 1 , para. 2: A Virtual Video Architecture); and assembling a virtual video 
using said synthetic character (fig. 1 , para. 2: A Virtual Video Architecture). 

Re claim 2, the method of claim 1 , wherein said step of extracting comprises the 
step of employing a stochastic background modeling technique to detect said object in 
said video stream. (See fig. 1 , para. 2: A Virtual Video Architecture). 

Re claim 3, the method of claim 1 , wherein said step of extracting comprises the 
step of employing a motion detection technique to detect said object in said video 
stream. (See fig. 1 , para. 2: A Virtual Video Architecture, also para 3: Detection of 
Moving Objects). 

Re claim 4, the method of claim 1 , wherein said step of extracting comprises the 
step of employing a segmentation technique to detect said object in said video stream. 
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(See fig. 1 , para. 2: A Virtual Video Architecture, also para 3: Detection of Moving 
Objects). 

Re claim 5, the method of claim 1 , wherein said step of extracting comprises the 
step of extracting background components from said video stream. (See fig. 1 , para. 2: 
A Virtual Video Architecture). 

Re claim 6, the method of claim 1 , wherein said step of extracting comprises the 
steps of: determining if a pixel in a frame of said video stream represents an object that 
is moving based on a stochastic model of a background scene in said frame; and 
clustering pixels in said frame that represent said object that is moving. (See fig. 1 , 
para. 2-5, in these segment, stochastic model of a background scene is discussed as 
well as clustering pixels). 

Re claim 7, the method of claim 1 , further comprising the step of tracking said 
object. (See para. 4: Tracking Targets). 

Re claim 8, the method of claim 7, wherein said step of tracking comprises the 
steps of: identifying a position of said object in a frame of said video stream; identifying 
candidate objects in a next frame of said video stream; and comparing said object in 
said frame with candidate objects in said next frame to determine a next position of said 
object in said next frame. (See para. 4-6). 

Re claim 9, the method of claim 1 , wherein said step of analyzing comprises the 
step of determining a rigidity of said object. (See para. 5: "Character Rigidity", also fig. 
4). 



Application/Control Number: 09/694,712 Page 18 

Art Unit: 2613 

Re claim 10, the method of claim 9, wherein said step of determining said rigidity 
comprises the step of determining a residual flow for said object. (See Lipton, para. 5: 
"Character Rigidity"). 

Re claim 1 1 , the method of claim 1 , wherein said step of analyzing comprises the 
step of determining a periodic sequence corresponding to said object. (See para. 6: 
"Determining a Periodic Sequence", also fig. 5). 

Re claim 12, the method of claim 11, wherein said periodic sequence represents 
one cycle of motion of said object over a series of frames, and wherein said periodic 
sequence comprises, for each frame of said set of frames, a visual appearance of said 
object and a frame-to-frame displacement of said object. (Lipton: para. 6: "Determining 
a Periodic Sequence", also fig. 5). 

Re claim 13, the method of claim 11, wherein said step of determining said 
periodic sequence comprises the steps of: collecting a set of visual templates of said 
object from a series of games of said video stream, said set of visual templates 
comprising at least one complete period of motion of said object; and matching a 
present visual template of said object with each visual template of said set of visual 
templates to determine a starting point of said period of motion of said object. (Lipton: 
para. 6: "Determining a Periodic Sequence", also fig. 5). 

Re claim 14, the method of claim 13, wherein said step of matching comprises 
the steps of: determining a convolution of said present visual template with each visual 
template of said set of visual templates; and selecting said starting point of said period 
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of motion based on a minimum of said convolution. (Lipton: para. 6: "Determining a 
Periodic Sequence", also fig. 5). 

Re claim 15, the method of claim 1, wherein said step of analyzing comprises the 
step of determining a periodic sequence corresponding to said object and at least one 
parameter describing an appearance of said object, and the method further comprising 
the step of transmitting or storing said periodic sequence and said at least one 
parameter. (See Lipton: para. 7, 7.1 and 8, in these segments, discussion of using 
periodic sequences of templates and velocities i.e. parameter to be captured and then 
replayed synthetically in arbitrary places and at arbitrary times from virtual video stream 
is indicative of the inherent need to store and transmit during real-time application). 

Re claim 16, the method of claim 1, wherein said step of manipulating is directed 
by a user. (See para. 1 , 8, these segments discuss a player interacts with a video 
game to manipulate analyzed object to obtain synthetic character). 

Re claim 17, the method of claim 1, wherein said step of manipulating is directed 
by a computation engine. (See para. 1: Introduction, in this segment, object analysis 
and synthetic character manipulation are computer controlled). 

Re claim 18, the method of claim 1, wherein said synthetic character is a hybrid 
based on said object of interest and computer-generated imagery. (See fig. 1, para. 1- 
2). 

Re claim 19, the method of claim 1, further comprising the step of providing a 
second synthetic character generated by a computer graphics engine, and wherein said 
step of assembling comprises the step of assembling said virtual video using said 
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synthetic character and said second synthetic character. (See figs. 1 , 5, para. 1-2, in 
fig. 5 for example, a template of multiple synthetic characters are available for 
selection). 

Re claim 20, the method of claim 1 , wherein said step of assembling comprises 
the step of inserting said synthetic character into said video stream. (See fig. 1 , para.1- 
2). 

Re claim 21 , the method of claim 20, wherein said synthetic character is inserted 
based on rigidity and periodicity of said synthetic character. (See para. 2: "A Virtual 
Video Architecture", this segment discusses several situations when it is necessary to 
insert synthetic characters into the virtual video stream in which rigidity and periodicity 
of a character are taken into account). 

Re claim 22, the method of claim 20, wherein said step of inserting comprises the 
steps of selecting a starting frame in said video stream and a position within said 
starting frame for inserting said synthetic character; and inserting a periodic sequence 
corresponding to said synthetic character beginning in said starting frame and at said 
position. (Lipton: para. 6: "Determining a Periodic Sequence", also fig. 5). 

Re claim 23, the method of claim 22, wherein said periodic sequence represents 
one cycle of motion of said synthetic character, and wherein said periodic sequence 
comprises, for each frame of said set of frames, a visual template of said synthetic 
character and a frame-to-frame displacement of said synthetic character. (Lipton: para. 
6: "Determining a Periodic Sequence", also fig. 5). 
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Re claim 24, the method of claim 23, wherein said step of inserting further 
comprises the step of multiplying each visual template by a scale factor to adjust a size 
of said synthetic character. (Lipton: para. 6: "Determining a Periodic Sequence", also 
fig. 5, also para 7: "Synthesizing Characters"). 

Re claim 25, the method of claim 23, wherein said step of inserting further 
comprises the step of multiplying each frame-to-frame displacement by a time factor to 
adjust a speed of motion of said synthetic character. (Lipton: para. 6: "Determining a 
Periodic Sequence", also fig. 5, also para 7: "Synthesizing Characters"). 

Re claim 26, the method of claim 23, wherein said step of inserting further 
comprises the step of applying a flip operator to each visual template to reverse a 
direction of motion of said synthetic character relative to a direction of motion of said 
synthetic character in said periodic sequence. (Lipton: para. 6: "Determining a Periodic 
Sequence", also fig. 5, also para 7: "Synthesizing Characters"). 

Re claim 27, the method of claim 20, wherein said synthetic character is 
occluded in said video stream. (Lipton: para. 7.1: "Synthesising Occluded Characters"). 

Re claim 28, the method of claim 20, wherein said step of inserting comprises the 
step of modifying at least one of appearance, scale, position, speed, direction of motion, 
and timing, prior to insertion of said synthetic character into said video stream. (Lipton: 
para. 6: "Determining a Periodic Sequence", also fig. 5, also para 7: "Synthesizing 
Characters"). 
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Re claim 29, the method of claim 1, wherein said step of assembling comprises 
the step of removing said synthetic character from said video stream. (See para. 1, 8, 
these segments discuss the ability to remove synthetic character from a video stream). 

Re claim 30, the method of claim 29, wherein said step of removing comprises 
the step of filling in pixels in frames of said video stream with pixels from a background 
model for said synthetic character removed from said video stream. (See para. 1-2, 8, 
these segments discuss the above limitations). 

Re claim 31 , the method of claim 29, wherein said step of removing comprises 
the step of repairing at least one of an uncovered background, a foreground object, and 
another synthetic character. (See para. 1-2, 8, these segments discuss the above 
limitations). 

Re claim 32, the method of claim 1 , further comprising the step of determining 
functional areas within said video stream. (See figs. 1 , 4-5, para. 2-6, these segments 
discuss the above limitations). 

Re claim 33, the method of claim 1, further comprising performing the method of 
claim 1 for a plurality of objects of interest in said video stream. (The limitations have 
been analyzed and rejected w/r to claim 32. In this segment, multiple objects are 
considered). 

Re claim 34, the method of claim 1, wherein said steps of extracting, analyzing, 
manipulating, and assembling are performed in real time. (See Abstract). 

Re claim 35, the method of claim 1 , wherein at least one of said steps of 
extracting, analyzing, manipulating, and assembling is performed in non-real time. 
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(Lipton discusses real-time application, however, capability to perform in non real-time 
is inherent). 

Re claim 36, the method of claim 1 , wherein said video stream comprises a 
background for a game. (See fig. 1 , para. 1-2). 

Re claim 37, the method of claim 1 , wherein said video stream comprises a 
simulation. (See Abstract, para. 1-2). 

Re claim 38, the method of claim 1 , wherein said video stream comprises a 
teleconference. (See para. 1). 

Re claim 39, the method of claim 1 , wherein said video stream comprises a 
distance education presentation. (See para. 1 , this segment discusses video- 
teleconferencing, which inherently has the capability of distance education 
presentation). 

Re claim 40, a computer system to perform the method of claim 1 . (See para. 1- 

2). 

Re claim 41 , a system comprising means for processing to perform the method 
of claim 1. (See para. 1-2). 

Re claim 42, a computer-readable medium comprising software to perform the 
method of claim 1. (See para. 1-2). 

Claims 43-58 have been analyzed and rejected w/r to claims 1-42 above. 
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Claim Objections 

10. Claims 45, 54 are objected to because of the following informalities: The 
identified claims have punctuation problems. They do not end with a period. 
Appropriate correction is required. 

Contact 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Vu Le whose telephone number is 703-308-6613. The 
examiner can normally be reached on M-F 8:30-6:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Chris Kelley can be reached on 703-305-4856. The fax phone number for 
the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



VffLe 
Primary Examiner 

AU 2613 
(703) 308-6613 
Vu.Le@uspto.gov 



